Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields.

Background: Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information technology academic literature sample real users or surrogate users is unknown. Therefore, it is uncertain whether or not the use of surrogate users in place of real users is a widespread problem within user study practice. Objective: To determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants. Method: We analyzed 725 user studies reported in 628 peer-reviewed articles published from 2013 through 2021 in 233 unique conference and journal outlets, retrieved from the ACM Digital Library, IEEE Xplore, and Web of Science archives. To study the sample selection choices, we categorized each study as generic (i.e., users are from the general population) or targeted (i.e., users are from a specific subpopulation), and the sampled study participants as real users (i.e., from the study population) or surrogate users (i.e., other than real users). Results: Our analysis of all 725 user studies shows that roughly two-thirds (75.4%) sampled real users. However, of the targeted studies, only around half (58.4%) sampled real users. Of the targeted studies sampling surrogate users, the majority (69.7%) used students, around one-in-four (23.6%) sampled through crowdsourcing, and the remaining 6.7% of studies used researchers or did not specify who the participants were. Conclusions: Key findings are as follows: (a) the state of sampling real users in information technology research has substantial room for improvement for targeted studies; (b) researchers often do not explicitly characterize their study participants in adequate detail, which is probably the most disconcerting finding; and (c) suggestions are provided for recruiting real users, which may be challenging for researchers. Implications: The results imply a need for standard guidelines for reporting the types of users sampled for a user study. We provide a template for reporting user study sampling with examples.

Persona preparedness: a survey instrument for measuring the organizational readiness for deploying personas.

User-centric design within organizations is crucial for developing information technology that offers optimal usability and user experience. Personas are a central user-centered design technique that puts people before technology and helps decision makers understand the needs and wants of the end-user segments of their products, systems, and services. However, it is not clear how ready organizations are to adopt persona thinking. To address these concerns, we develop and validate the Persona Readiness Scale (PRS), a survey instrument to measure organizational readiness for personas. After a 12-person qualitative pilot study, the PRS was administered to 372 professionals across different industries to examine its reliability and validity, including 125 for exploratory factor analysis and 247 for confirmatory factor analysis. The confirmatory factor analysis indicated a good fit with five dimensions: Culture readiness, Knowledge readiness, Data and systems readiness, Capability readiness, and Goal readiness. Higher persona readiness is positively associated with the respondents' evaluations of successful persona projects. Organizations can apply the resulting 18-item scale to identify areas of improvement before initiating costly persona projects towards the overarching goal of user-centric product development. Located at the cross-section of information systems and human-computer interaction, our research provides a valuable instrument for organizations wanting to leverage personas towards more user-centric and empathetic decision making about users. Supplementary Information: The online version contains supplementary material available at 10.1007/s10799-022-00373-9.

Big Data, Small Personas: How Algorithms Shape the Demographic Representation of Data-Driven User Segments.

Derived from the notion of algorithmic bias, it is possible that creating user segments such as personas from data results in over- or under-representing certain segments (FAIRNESS), does not properly represent the diversity of the user populations (DIVERSITY), or produces inconsistent results when hyperparameters are changed (CONSISTENCY). Collecting user data on 363M video views from a global news and media organization, we compare personas created from this data using different algorithms. Results indicate that the algorithms fall into two groups: those that generate personas with low diversity-high fairness and those that generate personas with high diversity-low fairness. The algorithms that rank high on diversity tend to rank low on fairness (Spearman's correlation: -0.83). The algorithm that best balances diversity, fairness, and consistency is Spectral Embedding. The results imply that the choice of algorithm is a crucial step in data-driven user segmentation, because the algorithm fundamentally impacts the demographic attributes of the generated personas and thus influences how decision makers view the user population. The results have implications for algorithmic bias in user segmentation and creating user segments that not only consider commercial segmentation criteria but also consider criteria derived from ethical discussions in the computing community.

Measuring user interactions with websites: A comparison of two industry standard analytics approaches using data of 86 websites.

This research compares four standard analytics metrics from Google Analytics with SimilarWeb using one year's average monthly data for 86 websites from 26 countries and 19 industry verticals. The results show statistically significant differences between the two services for total visits, unique visitors, bounce rates, and average session duration. Using Google Analytics as the baseline, SimilarWeb average values were 19.4% lower for total visits, 38.7% lower for unique visitors, 25.2% higher for bounce rate, and 56.2% higher for session duration. The website rankings between SimilarWeb and Google Analytics for all metrics are significantly correlated, especially for total visits and unique visitors. The accuracy/inaccuracy of the metrics from both services is discussed from the vantage of the data collection methods employed. In the absence of a gold standard, combining the two services is a reasonable approach, with Google Analytics for onsite and SimilarWeb for network metrics. Finally, the differences between SimilarWeb and Google Analytics measures are systematic, so with Google Analytics metrics from a known site, one can reasonably generate the Google Analytics metrics for related sites based on the SimilarWeb values. The implications are that SimilarWeb provides conservative analytics in terms of visits and visitors relative to those of Google Analytics, and both tools can be utilized in a complementary fashion in situations where site analytics is not available for competitive intelligence and benchmarking analysis.

Topic-driven toxicity: Exploring the relationship between online toxicity and news topics.

Hateful commenting, also known as 'toxicity', frequently takes place within news stories in social media. Yet, the relationship between toxicity and news topics is poorly understood. To analyze how news topics relate to the toxicity of user comments, we classify topics of 63,886 online news videos of a large news channel using a neural network and topical tags used by journalists to label content. We score 320,246 user comments from those videos for toxicity and compare how the average toxicity of comments varies by topic. Findings show that topics like Racism, Israel-Palestine, and War & Conflict have more toxicity in the comments, and topics such as Science & Technology, Environment & Weather, and Arts & Culture have less toxic commenting. Qualitative analysis reveals five themes: Graphic videos, Humanistic stories, History and historical facts, Media as a manipulator, and Religion. We also observe cases where a typically more toxic topic becomes non-toxic and where a typically less toxic topic becomes "toxicified" when it involves sensitive elements, such as politics and religion. Findings suggest that news comment toxicity can be characterized as topic-driven toxicity that targets topics rather than as vindictive toxicity that targets users or groups. Practical implications suggest that humanistic framing of the news story (i.e., reporting stories through real everyday people) can reduce toxicity in the comments of an otherwise toxic topic.

Mapping online hate: A scientometric analysis on research trends and hotspots in research on online hate.

Internet and social media participation open doors to a plethora of positive opportunities for the general public. However, in addition to these positive aspects, digital technology also provides an effective medium for spreading hateful content in the form of cyberbullying, bigotry, hateful ideologies, and harassment of individuals and groups. This research aims to investigate the growing body of online hate research (OHR) by mapping general research indices, prevalent themes of research, research hotspots, and influential stakeholders such as organizations and contributing regions. For this, we use scientometric techniques and collect research papers from the Web of Science core database published through March 2019. We apply a predefined search strategy to retrieve peer-reviewed OHR and analyze the data using CiteSpace software by identifying influential papers, themes of research, and collaborating institutions. Our results show that higher-income countries contribute most to OHR, with Western countries accounting for most of the publications, funded by North American and European funding agencies. We also observed increased research activity post-2005, starting from more than 50 publications to more than 550 in 2018. This applies to a number of publications as well as citations. The hotbeds of OHR focus on cyberbullying, social media platforms, co-morbid mental disorders, and profiling of aggressors and victims. Moreover, we identified four main clusters of OHR: (1) Cyberbullying, (2) Sexual solicitation and intimate partner violence, (3) Deep learning and automation, and (4) Extremist and online hate groups, which highlight the cross-disciplinary and multifaceted nature of OHR as a field of research. The research has implications for researchers and policymakers engaged in OHR and its associated problems for individuals and society.